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Setting  the  Stage 
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Setting  the  Stage 

•  The  need  for  “measuring”  operational  activities  &  their  effectiveness 

•  Are  we  doing  the  right  things? 

•  Are  we  using  the  right  tools  to  measure? 

•  Are  we  measuring  the  right  things? 
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Today’s  Operating  Environment 


Rapid  changes  in 
technology  and  its 
application  in  a  wide  range 
of  industries. 


Introduction  of  many  new 
systems,  business 
processes,  markets,  risks, 
and  enterprise  approaches. 


Many  immature  products 
and  services  being 
consumed  by  enterprises 
that  themselves  are  in  a 
state  of  change. 


(cm 
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Challenges  at  Hand 


How  can  you  tell  if  you  are  doing  a  good  job  of  managing  these  changes? 


How  best  to  monitor  your  progress  on  an  ongoing  basis? 


How  do  you  manage  the  interactions  of  systems 
and  processes  that  are  continually  changing? 


How  do  poor  processes  impact 
interoperability,  safety,  reliability, 
efficiency,  and  effectiveness? 


(ceict 
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Which  tool  should  I  use? 


Your  organization  wants  to  know  SOMETHING  about  your 
mission  operation: 

•  How  EFFECTIVE  are  we? 

•  Do  we  have  the  right  SKILLS  and  CAPABILITIES? 

•  Do  we  have  the  right  TECHNOLOGIES? 


(CEICT 
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Observation 


v  The  development  and  use 
of  maturity  models  in  security, 
continuity,  IT  operations,  & 
resilience  space  is  increasing 
dramatically. 


Software  Engineering  Institute  GarnegieMelbm 


Do  maturity  models  measure  the  right  thing? 


❖  May  not  measure  what  you  think  it  measures 

>  Practice  maturity  vs.  organizational  maturity? 

❖  May  give  you  inaccurate  data  on  which  to 
base  decisions 

>  Process  performance  vs.  product  performance? 

❖  Can  increase  cost  but  reduce  benefit 

>  An  improved  process  may  not  result  in  compliance 

❖  May  provide  a  false  sense  of  confidence 

>  A  robust  process  may  not  stop  all  malware 
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ABCs  of  Maturity  Models 

•  What  are  Maturity  Models? 

•  Types  of  Maturity  Models 

•  Examples  of  Maturity  Models 
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Maturity  Model  Defined 


An  organized  way  to  convey  a  path  of  experience, 
wisdom,  perfection,  or  acculturation. 

Depicts  an  evolutionary  progression  of  an 
attribute,  characteristic,  pattern,  or 
practice. 


The  subject  of  a  maturity  model  can  be 
objects  or  things,  ways  of  doing 
something,  characteristics  of 
something,  practices, 


controls,  or  processes. 


(CEICT 
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Maturity  Models  Provide... 


Means  for  assessing  and  benchmarking  performance 
Ability  to  assess  how  a  set  of  characteristics  have  evolved 
Expression  of  body  knowledge  of  best  practices 
Identification  of  gaps  and  improvement  plans 
Roadmap  for  model-based  improvement 
Demonstrated  results  of  improvement  efforts 
Common  language  or  taxonomy 
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Key  Components  of  a  Maturity  Model 


Levels 

•  The  measurement  scale 

•  The  transitional  states 


Domains 

•  Logical  groupings  of  like  attributes  into  areas  of  importance  to  the 
subject  matter  and  intent  of  the  model 

•  Logical  groupings  of  like  practices,  processes,  or  good  things  to  do 

Attributes 

•  Core  content  of  the  model  arranged  by  domains  and  levels 

•  Typically  based  on  observed  practices,  standards,  or  expert  knowledge 

Diagnostic  Methods 

•  For  assessment,  measurement,  gap  identification,  benchmarking 

Improvement  Roadmaps 

•  To  guide  improvement  efforts  (e.g.,  Plan-Do-Check-Act) 


(cm 
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Types  of  Maturity  Models 


There  are  three  types  of  maturity  models 

•  Progression  Maturity  Models 

•  Capability  Maturity  Models  (CMM) 

•  Hybrid  Maturity  Models 


One  or  more  may  be  appropriate 
for  your  particular  needs 


Not  all  maturity  models  are  CMMs 
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Progression  Maturity  Models 


Simple  progression  or  scaling  of  an 
characteristic,  pattern,  or  practice 


Levels  describe  higher  states 
of  achievement,  advancement, 
completeness,  or  evolution 


Levels  can  be  arbitrary  as 
agreed  upon  by  users, 
industry,  etc. 


attribute, 


(CEICT 
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Progression  Maturity  Models  -  Example 


A  Maturity  Progression  for 
Toy  Building  Bricks 

Lego  Mindstorms 

Lego  Architecture 

Lego  Technic 

Lego  City 

Lego  Duplo 
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Progression  Maturity  Models  -  Example 


A  Maturity 
Progression  for 
Human  Mobility 

Fly 

Sprint 

Run 

Jog 

Walk 

Crawl 


A  Maturity  Progression  for 
Authentication 

Three-factor  authentication 

Two-factor  authentication 

Addition  of  changing  every  60  days 

Use  of  strong  passwords 

Use  of  simple  passwords 


Progress  does  not  necessarily  equate  to  maturity 


(cm 
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Progression  Maturity  Models  -  Example 


Higher  levels  may  be 
characterized  as 
"tool-enabled" 


These 
characterizations 
are  typically 
arbitrary 

W 

Lower  levels  may  be 
characterized  as 
"primitive" 


A  Maturity 
Progression  for 
Counting 

Computer 

Calculator 

Adding  machine 

Slide  rule 

Abacus 

Pencil  and  paper 

Sticks/Stones 

Fingers 


(cm 
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Progression  Model  Example:  SGMM 


Smart  Grid  Maturity  Model 


SMR 

Strategy, 
Management,  & 
Regulatory 

OS 

Organization  & 
Structure 

GO 

Grid  Operations 

WAM 

Work  &  Asset 
Management 

TECH 

Technology 

CUST 

Customer 

VCI 

Value  Chain 
Integration 

SE 

Societal  & 
Environmental 
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Benefits  and  Limitations  of  Progression 
Models 


Benefits 

❖  Provides  a  transformative 
roadmap 

❖  Simple  to  understand  and 
adopt;  low  adoption  cost 

❖  Easy  to  recalibrate  as 
technologies  and  practices 
advance 


Limitations 

❖  Levels  are  arbitrarily  defined 
and  may  be  meaningless 

❖  Achieving  higher  levels  does 
not  necessarily  translate  into 
“maturity” 

❖  Often  confused  with  CM  Ms — 
thus  users  inaccurately  project 
traits  of  CM  Ms  on  progression 
models 


(cm 
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Capability  Maturity  Models  (CMM) 


A  more  complex  instrument 


Characterizes 


•  the  maturity  of  processes 

•  the  degree  to  which  processes  are  institutionalized 

•  the  degree  to  which  the  organization  demonstrates  process  maturity 


•  the  maturity  of  the  culture  of  the  organization 

Levels  reflect  the  degree  to  which  a  particular 
set  of  practices  have  been  institutionalized 

•  Institutionalized  processes  are  more  likely 
to  be  retained  during  times  of  stress. 


(cm 
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What  do  these  organizations  have  in 
common? 


Tradition 

Protection 


Customer  Happiness 


Chain  of  Command 
Unit  Cohesion 
Regulations 


fcE^ 
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CMM  Levels  -  An  Example 


i 

i 

i 

i 

i 

i 

i 

i 

i 


Processes  are  1 
acculturated,  \ 
defined J 
measured '  \ 
and 1 

i 

governed  i 


Practices  are  1 
i  performed , 


Practices  are  1 

1  i 

i  incomplete  \ 


-  Level  0 
•  Incomplete 


Higher  degrees  of 
institutionalization 
translate  to  more  stable 
processes  that 

•  are  repeatable 

•  produce  consistent 
results  over  time 

•  are  retained  during 
times  of  stress 
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Examples  of  CMM  Levels 


Example  1 

Optimized 

Quantitatively  Managed 

Defined 

Managed 

Ad  hoc 


Example  2 

Externally  integrated 
Internally  integrated 
Managed 
Performed 
Initiated 
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Shared 

Defined 

Measured 

Managed 

Planned 

Performed  but  ad  hoc 
Incomplete 
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Capability  Maturity  Model  Example:  CERT-RMM 


CERT  -RMM,  VERSION  1.1 


CERT  Resilience 
Management  Model 


\\\  I 


& 


m 


A  Maturity 
Model  for 
Managing 
Operational 
Resilience 


Framework  for 
managing  and  improving 
operational  resilience 


Richard  A.  Caralli 
Julia  H.  Allen 
David  W.  White 


http://www.cert.org/resilience/ 


"...an  extensive  super-set  of 
the  things  an  organization 
could  do  to  be  more  resilient  ” 


-  CERT-RMM  adopter 


fcE^ 
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CMM  Example:  CERT-RMM 


CERT-RMM  Process  Areas  (Domains) 


Access  Management 

Measurement  and  Analysis 

Asset  Definition  and  Mgmt. 

Monitoring 

Communications 

Organizational  Process  Focus 

Compliance 

Organizational  Process  Definition 

Controls  Management 

Organizational  Training  &  Awareness 

Enterprise  Focus 

People  Management 

Environmental  Control 

Resilience  Requirements  Development 

External  Dependencies 

Resilience  Requirements  Mgmt. 

Financial  Resource  Mgmt. 

Resilient  Technical  Solution  Engr. 

Human  Resource  Management 

Risk  Management 

Identity  Management 

Service  Continuity 

Incident  Management  &  Control  Technology  Management 

Knowledge  &  Information  Mgmt.  Vulnerability  Analysis  &  Resolution 


fcE^ 
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CMM  Example:  CERT-RMM 


Consider  the  Incident  Management  and  Control  (IMC) 
domain  from  CERT-RMM: 


Goal  1:  Establish  the  IMC  process 

Goal  2:  Detect  events 

Goal  3:  Declare  incidents 

Goal  4:  Respond  to  and  recover  from  incidents 

Goal  5:  Establish  incident  learning 


(cm 
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CMM  Example:  CERT-RMM 


"We  don't  do 
(all  of)  the 
practices." 


"We  do  the 
practices." 


Institutionalization  is  cumulative 


"We  do  the 
practices  AND 
we  plan  and 
govern  the 
process, 
resource  it, 
train  people 
to  do  it, 
monitor  it, 
etc..." 


We  do 

everything  in 
level  2  AND 
we  have  a 
defined 
process  and 
collect 

improvement 

information." 
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Benefits  and  Limitations  of  CMMs 


Benefits 

❖  Provides  for  measurement 
of  core  competencies 

❖  Provides  for  rigorous 
measurement  of 
capability — the  ability  to 
retain  core  competencies 
under  times  of  stress 

❖  Can  provide  a  path  to 
quantitative  measurement 


Limitations 

❖  Sometimes  difficult  to 
understand  and  apply;  high 
adoption  cost 

❖  “Maturity”  may  not  translate  into 
actual  results 

❖  Potential  false  sense  of 
achievement:  achieving  high 
maturity  in  security  practices 
may  not  mean  the  organization 
is  “secure” 


(cm 
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Compare:  Progression  vs  CMM 


Level  3 

•  Run 


—  Level  2 

•  Jog 

—  Level  1 

•  Walk 

—  Level  0 

•  Crawl 


Progression  Model 


—  Level  3 


Defined 


^  Level  2 


—  Level  1 


Performed 


i—  Level  0 


Core  practices 


Managed 


Incomplete 


o 

.ns 

o  c 

+J  o 

S  ^ 

Q  -S 


Capability  Model 


fcE^ 
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Hybrid  Maturity  Model 


Combines  the  best  features  of  progression  and  capability 
maturity  models 

•  Allows  for  measurement  of  evolution  or  achievement  as  in 
progression  models 

•  Adds  the  ability  to  measure  capability  or  institutionalization  with  the 
rigor  of  a  CMM 


Levels  reflect  both  achievement  and  capability 
Transitions  between  levels: 

•  Similar  to  a  capability  model 
(i.e.,  describe  capability  maturity) 

•  Architecturally  use  the  characteristics,  indicators, 
attributes,  or  patterns  of  a  progression  model 
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Capability  or  "maturity"  levels 


Hybrid  Maturity  Models 


Domain  1 


Level  4 

Defined 

Level  3 

Measured 

Level 

Manageo 


Domains:  Specific  categories  of 
attributes,  characteristics,  patterns,  or 
practices  that  form  the  content  of  the 
model 


Level  1 

Planned 

Level  0 

Incomplete 

Maturity  levels:  Defined  sets  of 
characteristics  and  outcomes,  plus 
capability  considerations 

^  _  Software  Engineering  Institute 

Carnegie  Mellon 

Model  content:  Specific  attributes, 
characteristics,  patterns,  or  practices 
that  represent  progression  and 
capability 
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Maturity  Indicator  Levels 


Hybrid  Model  Example:  ES-C2M2 


10  Model  Domains:  Logical  groupings  of  cybersecurity  practices 


ELECTRICITY  SUBSECTOR 

CYBERSECURITY  CAPABIUTY  MATURITY  MODEL  (ES-C2M2) 


31  M*  2012 


Electricity 

Subsector 

Cybersecurity 

Capability 

Maturity 

Model 

(ES-C2M2) 


fcE^ 
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Benefits  and  Limitations  of  Hybrid  Models 


Benefits 


Limitations 


❖  Provides  for  easy 
measurement  of  core 
competencies  as  well  as 
approximation  of  capability 

❖  Can  adapt  easily  to 
evolution  of  technologies 
and  practices  without 
sacrificing  capability 
measurement 

❖  Low  adoption  cost 


❖  “Maturity”  concept  is 
approximated;  not  as  rigorous 
as  CMM 

❖  Combination  of  attributes  with 
institutionalizing  features  at 
each  level  can  be  arbitrary 


(cm 


Software  Engineering  Institute  CarnegieMelkin 


©2014  Carnegie  Mellon  University 


35 


Closing  Thoughts 

•  A  few  cautions 

•  Determining  when  and  which  type  to  use 


^  r 

/ i 


fcE^ 
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First  and  Foremost 


Have  a  clear  understanding  of  your  business  objectives  for 
using  any  type  of  improvement  model 

•  How  the  model  will  meet  these  objectives 

Understand  how  this  initiative  fits  with  others  that  are 
mainstream  for  the  organization  (not  a  new  add-on) 

Have  visible  sponsorship  of  executives  and  senior  leaders 
who  are  essential  for  success 

Have  well-defined  outcome  measures  that  are  regularly 
reported  and  reviewed 

Have  a  plan  and  committed  resources 
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A  Few  Cautions 


Progression  models  may  be  easier  to  adopt  but  may  not  be 
sustainable  (aka  sticky) 

Definitions  of  levels  can  be  arbitrary 

Measuring  process  performance  and  maturity  is  useful  but 
may  not  be  sufficient 


Exercise  care  when  using  maturity  models  for  specific 
purposes 


(CEICT 
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When  Does  It  Make  Sense  to  Use  Maturity 
Models? 


Requirement  for  a  structured  approach 

Demonstrated,  measurable  results  based  on  an  established 
body  of  knowledge 

A  defined  roadmap  from  a  current  state  to  a  desired  state 

An  ability  to  monitor  and  measure  progress,  particularly  in  the 
presence  of  change 

•  Response  to  a  strategic  improvement  or  new  product/new  market 
objective 
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When  Does  It  Make  Sense  to  Use  Maturity 
Models?  (cont.) 


Desire  to  answer  these  questions  in  a  repeatable,  predictable 
manner: 

•  How  do  I  compare  with  my  peers?  (ability  to  benchmark) 

•  How  can  I  determine  how  secure  I  am  and  if  I  am  secure  enough? 

•  How  do  I  measure  my  current  state?  Characterize  my  desired  state? 

•  What  concrete  actions  do  I  need  to  take  to  improve?  And  in  what 
order? 

•  How  do  I  measure  progress  toward  my  desired  state? 

•  How  do  I  adapt  to  change? 
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Thank  you  for  your  attention... 
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